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Abstract 


Acquisition  research  has  recently  laid  emphasis  on  the  study  of  the  cascading 
effects  of  interdependencies  in  the  joint  space  of  Major  Defense  Acquisition 
Programs  (MDAPs).  We  are  interested  in  proactively  modeling  the  non-linear 
cascading  effects  of  interdependencies  in  highly  dependent  networks.  Specifically, 
in  this  report  we  examine  DoD  acquisition  in  the  context  of  when  MDAPs  exchange 
and  share  resources  for  the  purpose  of  establishing  joint  capabilities.  Our 
hypothesis  is  that  examining  the  interdependent  regions  among  MDAPs  from 
multiple  perspectives  using  non-linear  methods  will  allow  for  “what-if”  analyses  and 
will  help  decision-makers  gain  insight  on  the  cascading  effects  of  perturbations  and 
take  appropriate  measures  to  handle  them.  Additionally,  we  also  ascertain  whether  a 
popular  decision-theoretic  model  for  decision-making  and  planning  for  cascading 
effects  in  the  face  of  uncertainty  is  appropriate  to  study  the  cascading  effects  among 
MDAPs.  Our  approach  is  to  use  a  case  study  to  determine  whether  the  data 
required  to  build  an  effective  decision-theoretic  model  is  available.  We  also  capture 
the  data  investigation  process  and  identify  the  challenges  that  were  encountered. 

Our  results  show  that  it  is  possible  to  recast  the  study  of  cascading  effects  in  MDAPs 
as  a  sequential  decision  problem.  We  describe  the  informational  value  in  the  existing 
data  and  challenges  inherent  in  the  data  collection  process. 

Keywords:  Major  Defense  Acquisition  Programs  (MDAPs),  non-linear 
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I.  The  Joint  Space  of  Major  Defense  Acquisition 
Programs  Networks 

It  has  been  shown  that  data  are  the  foundation  for  decision-making  in  the 
acquisition  environment.  The  DoD  has  spent  a  significant  amount  of  effort  working 
across  the  organization  to  identify  useful  sources  of  data  and  to  conduct  analyses. 
The  importance  to  acquisition  research  of  studying  Major  Defense  Acquisition 
Programs’  (MDAPs)  interdependencies  was  emphasized  during  the  2012  Annual 
Acquisition  Research  Symposium  by  the  introduction  of  a  new  panel  titled  Predicting 
Performance  and  Interdependencies  in  Complex  Systems  Development.  Prior 
research  has  established  that  MDAPs  are  demonstrably  interdependent  and  that 
they  can  be  thought  of  as  networks  of  interdependent  programs  (Lewin,  1999; 

Flowe,  Brown,  &  Hardin,  2009).  Also,  the  acquisition  paradigm  established  in  statute 
(10  U.S.C.  2434;  Defense  Acquisition  Workforce  Act,  1990),  in  policy  (DoD  5000.02; 
Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  [USD(AT&L)j, 
2008),  and  in  regulation  tends  to  favor  the  notion  of  MDAPs  as  being  independent, 
which  would  cause  exogenous  factors  caused  by  interdependence  to  be  overlooked 
or  misinterpreted. 

Although  it  is  critically  important  to  understand  the  program  interfaces  and 
interdependencies,  there  are  few  tested  and  proven  tools  for  program  managers  and 
acquisition  executives  to  probe  the  joint  space  or  to  track  the  cascading  effects  that 
the  joint  space  might  trigger.  There  is  reason  to  believe  that  the  exogenous  issues 
generated  from  the  shared  domains  remain  unnoticed  to  the  extent  of  causing  the 
program  to  potentially  experience  severe  performance  degradation  (Brown,  201 1). 
The  complexity  of  the  joint  environment  is  likely  to  have  a  direct  bearing  on 
acquisition  activities.  The  precise  effect  on  acquisition,  and  its  resulting  managerial 
implications,  are,  as  of  yet,  unknown.  We  believe  that  given  the  frequency  with  which 
government  agencies  are  moving  toward  joint  initiatives,  the  findings  of  this  research 
project  based  on  DoD  programs  may  prove  instrumental  to  a  wide-ranging  audience. 
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Furthermore,  at  the  2012  Acquisition  Symposium,  Dr.  Frank  Kendall  III,  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]), 
discussed  the  DoD’s  strategic  priorities,  especially  around  acquisition.  These 
priorities  included  achieving  affordable  programs  that  execute  well  and  improving 
efficiency  (via  Better  Buying  Power  and  other  initiatives). 

Along  with  other  researchers  (Brown  &  Owen,  2012),  we  have  begun  to 
harness  a  network-centric  approach  to  study  DoD  acquisition  and  focus  on  an 
MDAP  network  of  interrelated  programs  that  exchange  and  share  resources  for  the 
purpose  of  establishing  joint  capabilities.  Some  work  (Zhao,  Gallup,  &  MacKinnon, 
2012)  has  been  done  to  analyze  the  unstructured  and  unformatted  acquisition 
program  data  using  a  data-driven  automation  system  called  Lexical  Link  Analysis 
(LLA).  LLA  is  used  to  determine  the  correlation  between  system  interdependency 
and  development  costs  in  an  effort  to  enable  acquisition  researchers  and  decision¬ 
makers  to  recognize  important  connections  that  form  patterns  derived  from  dynamic 
data  collection.  In  other  work  (Han,  Fang,  &  DeLaurentis,  2012),  a  Bayesian  Network 
(BN)  method  is  used  to  assess  the  cascading  effects  of  requirement  and  systems 
interdependencies  on  risk  in  an  effort  to  effectively  analyze  alternatives  in  a 
capability-based  acquisition  strategy.  The  technique  is  evaluated  within  a  synthetic 
network  and  identifies  critical  systems  and  requirements. 

This  research  seeks  to  understand  and  model  the  behavior  of  non-linear 
cascading  effects  in  the  joint  space  of  MDAPs,  where  their  transactions  form 
interdependencies.  The  transactional  flows  in  and  out  of  MDAPs  collectively  form 
networks  of  interdependent  programs  that  can  be  examined  to  conduct  scenario 
planning  or  “what-if”  analyses.  These  what-if  analyses  will  help  decision-makers  gain 
insight  on  the  cascading  effects  of  perturbations  and  take  appropriate  measures  to 
handle  them.  We  develop  models  that  can  address  what-if  scenarios  such  as  the 
following:  What  if  my  partner  reneges  on  a  funding  obligation?  What  if  Congress 
alters  my  funding?  How  will  the  perturbation  affect  my  partners?  The  research  also 
identifies  and  enumerates  the  characteristics  in  the  existing  MDAP  data  that  are 
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critical  to  building  a  complete  model  of  MDAP  behavior  and  discusses  the 
challenges  in  acquiring  some  of  these  data  so  that  appropriate  governance 
mechanisms  can  then  be  isolated.  This  data  acquisition  process  is  emphasized  as 
much  as  the  behavioral  findings,  with  the  hope  that  the  lessons  learned  from  the 
process  will  allow  for  more  accurate  and  complete  data  gathering  and  modeling  in 
future  iterations  of  this  work. 

The  MDAP  data  that  we  analyze  include  Selected  Acquisition  Reports  (SAR), 
Defense  Acquisition  Executive  Summaries  (DAES),  and  Program  Element  (PE) 
documents  over  multiple  years.  Although  our  aim  is  to  work  on  the  entire  collection 
of  MDAPs,  we  observe  that  this  eclectic  conglomeration  of  information  is  highly 
unstructured,  significantly  inordinate,  and  unmanageably  colossal  for  manual 
analyses.  Hence  we  focus  on  a  case  study  that  contains  a  small  set  of  existing 
MDAPs.  We  use  fictitious  names  (e.g.,  MDAP_A,  MDAP_B,  etc.)  to  retain 
confidentiality  of  individual  program  information.  In  this  case  study,  we  do  an  in- 
depth  analysis  of  the  data  and  study  their  complex  interrelationships  from  multiple 
perspectives  with  the  hope  that  some  of  our  observations  and  lessons  learned  about 
MDAPs  and  the  analysis  process  can  then  be  scaled  to  the  entire  network. 

We  study  whether  performance  breaches  correlate  with  interdependency 
characteristics  in  the  context  of  a  small  MDAP  network.  As  a  consequence  of  this 
work,  in  future  studies  we  can  model  the  effects  of  acquisition  decisions  or  program 
outcomes,  such  as  a  breach  at  a  node  or  resource  cutoff  in  some  in-flow,  and  predict 
their  likely  effects.  We  can  extend  that  to  conjunctures  of  breaches  or  breaks  in  the 
flows.  Similarly,  we  can  determine  the  most  robust  and  weakest  programs  in  the 
system  (i.e.,  those  most  and  least  likely  to  have  breaches  or  fail).  We  also  can  use 
the  model  to  examine  the  changes  to  the  system  that  might  increase  its  robustness. 

The  significance  of  the  research  is  three-fold: 

■  It  aims  to  forge  new  ground  in  identifying  the  effects  of 

interdependency  on  acquisition  and,  if  needed,  uncovering  early 
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indicators  of  interdependency  risk  so  that  appropriate  governance 
oversight  methods  can  then  be  isolated. 

■  It  provides  insight  into  the  nature  of  the  available  data  and  whether 
they  can  support  the  use  of  non-linear  methods  to  detect  and  prevent 
cascading  consequences. 

■  It  verifies  using  a  small  use-case  the  viability  of  a  decision-theoretic 
model  to  describe  the  sequential  decision-making  process  inherent  to 
MDAPs.  This  model,  which  will  be  implemented  in  future  work,  will 
capture  uncertainty  in  action  outcomes  and  information  about 
neighboring  nodes. 

A.  Research  Methodology 

To  perform  this  study,  we  designed  a  methodology  with  four  goals.  We  first 
selected  a  small  subset  of  inter-related  MDAPs  based  on  a  set  of  criteria  to  form  our 
case  study.  We  defined  Goals  1  and  2  to  determine  whether  the  MDAP  data  in  the 
form  of  the  SARs,  DAES,  and  PEs  is  sufficient  to  identify  the  effects  of 
interdependency  on  acquisition  and  uncover  any  early  indicators  of  interdependency 
risk.  These  goals  also  determined  whether  a  decision-theoretic  model  in  Goal  3  is  a 
feasible  next  step.  Having  verified  that  this  is  the  case,  we  then  formulated  a 
decision-theoretic  model.  Finally,  we  captured  the  essence  of  the  data  acquisition 
process  for  our  study  and  the  lessons  learned. 

Goal  1:  Identify  highly  dependent  parts  of  the  MDAP  network: 

■  What  are  the  essential  features  of  the  network  that  reveal  the  joint 
space  dynamics? 

■  What  are  the  relative  priorities  associated  with  these  features  and  how 
do  they  affect  the  network  relationship? 

Goal  2:  Analyze  and  understand  the  data  available  from  MDAP  performance 
reports  to  extract  features  of  network  dynamics: 

What  are  the  local  issues  that  lead  toward  breach  or  near-breach 
situations? 
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How  often  and  why  do  the  local  mitigation  efforts  fail  to  improve 
performance? 


■  How  do  we  identify  the  non-local  issues  that  result  from  the 
interdependencies? 

■  Can  we  determine  the  cascading  effect  through  the  network? 

Recognizing  that  most  acquisition  decisions  are  framed  by  the  assumption  of 
programmatic  independence,  exogenous  effects  are  not  well  represented  in 
acquisition  policy  or  practice.  Therefore,  we  approached  Goal  2  from  two 
perspectives:  a  local  perspective  in  which  the  analyses  are  based  solely  on  the 
individual  program’s  own  data,  and  a  non-local  perspective  in  which  the  analyses 
are  based  on  the  data  of  MDAPs  existing  in  the  joint  space  of  the  individual  program. 
We  believe  lessons  learned  from  these  analyses  should  enable  the  stakeholders  to 
take  appropriate  measures  to  improve  the  performance  of  the  programs. 

Goal  3:  Identify  the  building  blocks  necessary  for  a  decision-theoretic  model 
that  harnesses  the  Decentralized-Markov  Decision  Process  (DEC-MDP)  formalism 
(this  model  will  be  implemented  in  future  work): 

■  What  are  the  essential  characteristics  of  the  MDAP  network  that  justify 
a  DEC-MDP  model? 

■  How  can  the  MDAP  network  be  modeled  as  a  decentralized  system? 

■  What  are  the  key  challenges  in  the  design  of  the  DEC-MDP? 

■  What  essential  features  should  the  DEC-MDP  model  incorporate  for 
better  predictability? 

The  DEC-MDP  is  a  sub-class  of  decentralized,  partially  observable  MDP 
(DEC-POMDP;  Bernstein,  Givan,  Immerman,  &  Zilberstein,  2002)  that  we  propose  to 
use  to  model  the  behavior  of  the  MDAP  network.  A  state  is  a  snapshot  in  time  of  the 
MDAP’s  status  that  consists  of  crucial  local  and  non-local  information.  A  policy  is  a 
mapping  from  a  state  to  an  action.  This  formalism  would  allow  the  MDAP  to  execute 
the  appropriate  local  policy  to  achieve  higher  performance.  Our  aim  is  to  define  a 
computationally  tractable  model. 
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Goal  4:  Understand  the  characteristics  of  the  existing  data  resources: 


■  What  are  the  challenges  to  pre-process  the  existing  data? 

■  What  key  information  do  we  gain  from  the  existing  data? 

■  What  are  the  key  limitations  in  the  existing  data? 

■  What  are  the  most  relevant  authoritative  data  sources? 

■  What  subset  of  this  data  is  analytically  useful? 

■  What  are  the  data  requirements  to  design  a  complete  DEC-MDP 
model? 

■  How  can  the  various  program-related  documents  be  integrated  in  a 
coherent  and  meaningful  fashion  to  aid  decision-makers  as  well  as 
researchers  to  build  complete  models? 

Goal  4  recommends  what  should  be  done  to  capture  information  so  that  the 
decision-making  process  becomes  efficient  and  complete.  It  also  opens  the 
discussion  to  alternative  sources  of  data  and  the  notion  that  the  DoD  collects  data 
that  could  be  improved  for  analytic  purposes. 

B.  Summary  of  Findings 

Our  findings  indicate  that  MDAP-related  data  characteristics  support  the 
multiple-perspective  study  of  perturbations,  and  it  should  be  possible  to  recast  the 
study  of  cascading  effects  as  a  sequential  decision  problem.  We  also  note  that  when 
using  manual  analysis  it  is  crucial  to  consider  the  uncertainty  in  action  outcomes  in 
the  decision-making  process  and  that  a  non-local  perspective  may  help  explain  a 
performance  breach  in  situations  where  a  solely  local  perspective  does  not.  We 
observe  that  the  conditions  that  apply  to  the  acquisition  oversight  and  management 
context  are  particularly  well  suited  to  future  analysis  using  decision  theory, 
specifically  sequential  decision-making  based  on  incomplete  information,  the  need  to 
support  various  possible  outcomes  over  time  to  support  effective  decision-making, 
the  availability  of  time-series  data  to  help  build  the  MDP  model,  etc.  This  verifies  our 
conjecture  that  decision  theory  is  a  good  avenue  to  study  interdependencies  in  the 
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MDAP  network  and  to  capture  early  indicators  of  interdependency  risk.  This  work 
also  identifies  the  need  to  have  access  to  large-scale  data  (of  many  MDAPs  over 
several  years)  to  build  an  accurate  full-scale  decision-theoretic  model.  This  is  the 
basis  for  our  proposed  research  for  the  next  phase  of  this  project  where  we  plan  to 
design  tools  for  large-scale  automated  batch  processing  of  MDAP  data  using  text 
and  image  analysis.  Finally,  we  have  captured  the  informational  value  in  the  existing 
data  and  challenges  inherent  in  the  data  collection  process  with  respect  to  their  role 
in  isolating  risks  and  initiating  appropriate  government  oversight  methods. 

The  rest  of  the  paper  is  structured  as  follows:  We  first  identify  the  network 
dependencies  among  the  MDAPs  and  define  a  sample  network  for  analyses  (Goal 
1);  we  then  investigate  the  local  and  non-local  causes  for  degradation  in 
performance  of  the  nodes  in  the  sample  network  (Goal  2);  then  we  present  the  DEC- 
MDP  model  formulation  (Goal  3)  followed  by  observations  made  about  the 
characteristics  of  the  available  data  (Goal  4);  and,  finally,  we  conclude  with  the 
lessons  that  we  learned  through  this  process. 
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II.  Network  Model 


In  this  section,  we  first  describe  various  MDAP  performance  reports  and 
discuss  their  significance  in  the  light  of  networking  dependencies  among  the 
MDAPs.  We  also  define  a  sample  funding  network  from  our  chosen  MDAPs  in  an 
effort  to  investigate  its  performance.  Specifically,  we  define  a  process  to  choose  an 
MDAP  program  to  be  the  focus  of  our  investigation  and  identify  its  immediate 
network  based  on  the  interrelationships  it  maintains  with  neighboring  MDAPs.  We 
will  use  the  lessons  learned  from  the  analyses  of  this  sample  network  to  build  an 
accurate  decision-theoretic  model. 

A.  Available  Data  Resources  on  MDAP  Performance 

The  information  pertaining  to  acquisition  research  is  overwhelming  and 
multifarious.  It  appears  to  be  a  daunting  task  for  acquisition  researchers,  let  alone 
program  managers,  to  integrate  and  understand  the  vast  and  dynamic  data  in  a 
coherent  way.  We  use  the  following  set  of  data  sources  to  define  the 
interrelationships  and  dependencies  among  the  MDAPs  from  a  network-centric 
viewpoint: 

■  Monthly  DAES  reports  that  provide  an  early-warning  report  on  the 
status  of  some  program  features  such  as  cost,  schedule,  performance, 
funding,  and  so  forth; 

■  SARs  that  summarize  the  latest  estimates  of  cost,  schedule,  and 
technical  status  to  be  reported  annually  in  conjunction  with  the 
president's  budget;  and 

■  R-docs  used  to  justify  the  congressional  budgeting  process. 

B.  Types  of  Interdependent  Networks  Within  the  MDAP 
Domain 

In  addition  to  the  above  data  sources,  program  managers  also  report  on  four 
external  interdependencies:  (1)  data  interdependencies  with  other  DoD  programs, 
(2)  funding  received  from  other  DoD  programs,  (3)  contractor  interdependencies, 
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and  (4)  budgeting/spending  authority  interdependencies.  This  information  is  useful  in 
identifying  four  types  of  interdependency  networks  among  the  MDAPs.  As  an 
example,  Figure  1  depicts  989  interdependencies  of  various  types  (resource, 
material,  authority,  etc.)  that  were  extracted  from  a  DAES  report  describing  the  78 
MDAPs  in  2010.  We  claim  that  the  existence  of  a  link  between  one  program  and 
another  provides  a  fairly  robust  basis  for  asserting  there’s  some  sort  of 
interdependence,  and  for  building  a  hypothetical  program  network  for  the  purposes 
of  the  analysis.  We  also  note  that  MDAP  programs  will  have  interdependencies  with 
non-MDAP  programs.  For  example,  the  Joint  Strike  Fighter  program  identifies  data 
and  funding  interdependencies  with  the  Italian,  German,  and  French  defense 
departments.  In  the  current  dataset,  17%  of  the  interdependencies  are  outbound, 
37%  are  inbound,  and  45%  are  bidirectional.  As  such,  the  flows  in  and  out  of  a  node 
can  be  examined  to  conduct  scenario  planning  or  what-if  analyses. 


Figure  1.  MDAP  Interdependencies  in  2010 

C.  Case  Study  of  MDAP_A  Funding  Network 

We  choose  to  do  a  case  study  because  of  the  characteristics  of  the  data. 
MDAP_A,  a  communications  program  initiated  in  2004  whose  program  name  has 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  - 10  - 

NAVAL  POSTGRADUATE  SCHOOL 


been  scrubbed  for  confidentiality  purposes,  is  the  central  MDAP  for  our  study.  This 
program  is  our  focus  because  (a)  the  data  available  about  this  program  are 
significant,  and  (b)  between  the  years  2006  to  2010,  it  experienced  multiple 
Acquisition  Program  Baseline  (APB)  breaches  and  percentage  increases  in  Program 
Acquisition  Unit  Cost  (PAUC),  making  it  a  critical  node  for  reference.  PAUC  is 
defined  as  the  ratio  of  the  current  Program  Acquisition  Cost  to  the  Program 
Acquisition  Quantity.  PAUC  combines  both  the  research,  development,  test  & 
evaluation  (RDT&E)  appropriation  (for  the  engineering  development  of  the  program) 
as  well  as  the  Procurement  appropriation  (for  the  production  of  the  system),  and  is 
sensitive  to  both  changes  in  development  and  production  funding  as  well  as 
changes  in  the  number  of  units  procured.  This  makes  it  a  particularly  worthwhile 
metric  of  overall  program  performance. 

Using  information  about  the  funding  partners  of  MDAP_A,  we  define  a  logical 
funding  network  shown  in  Figure  2.The  other  nodes  in  the  graph  are  neighbor 
programs  of  MDAP_A  that  share  common  funding  agencies.  The  funding  network  is 
defined  based  on  the  number  of  PEs  that  funded  the  MDAP  RDT&E  efforts.  PE  is 
the  code  number  assigned  by  the  comptroller.  Since  PEs  fund  multiple  MDAPs, 
programs  that  share  a  common  PE  monitor  could  be  isolated.  Procurement  PEs 
were  not  considered  for  defining  funding  networks  since  the  RDT&E 
interdependencies  were  the  most  critical  to  program  performance.  The  funding 
network  and  the  associated  R-docs  allowed  us  to  do  a  detailed  study  of  the 
performance  of  the  member  nodes  and  to  understand  the  cascading  effects.  In  the 
future,  we  plan  to  apply  the  lessons  learned  from  this  focused  study  to  the  entire 
MDAP  network. 


NAVAL  POSTGRADUATE  SCHOOL 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 


- 11  - 


Figure  2.  Funding  Network  of  MDAP  A 


We  analyzed  the  data  that  we  gathered  from  the  available  performance 
reports  of  all  the  MDAPs  in  the  MDAP_A  funding  network,  from  local  and  non-local 
perspectives,  as  discussed  in  the  next  section.  Consider  the  funding  network  for 
MDAP_A  in  Figure  2.  MDAP_A  lies  at  the  center  of  this  undirected  network  that 
contains  five  nodes.  The  link  between  any  two  nodes  refers  to  the  funding 
relationship  and  represents  an  interdependency  between  the  programs.  These  links 
illustrate  the  interdependent  regions  of  the  case  study  network.  We  analyzed  the 
performance  of  the  programs  based  on  the  APB  breaches  and  amount  of  increase  in 
%PAUC,  which  is  the  percentage  change  of  the  current  program  acquisition  unit  cost 
estimate  relative  to  the  original  APB  value.  Programs  for  which  the  current  estimate 
of  PAUC  has  increased  by  15%  or  more  over  the  currently  approved  APB  must 
report  a  unit  cost  breach  to  the  congressional  defense  committees.  Five  types  of 
APB  breaches  are  reported  in  the  performance  reports:  schedule,  performance, 
RDT&E,  procurement,  and  PAUC.  A  program  is  considered  to  perform  poorly  if  it 
experiences  frequent  APB  breaches  and/or  increases  in  %PAUC. 

We  note  that  the  central  program  MDAP_A  has  been  underperforming  for  a 
period  of  time,  and  we  also  note  that  some  neighboring  programs  have  been 
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underperforming  in  subsequent  periods.  We  want  to  understand  their  performance 
degradation  by  investigating  the  following  questions: 


Q1 :  What  are  the  local  reasons  for  a  program  (e.g.,  MDAP_A)  to 
underperform? 

Q2:  How  often  and  why  the  forecasting  of  mitigation  efforts,  as 
captured  in  monthly  DAES  reports,  turns  out  to  be  ineffective? 

Q3.  What  are  the  non-local  reasons  for  poor  performance? 

Q4:  How  does  the  effect  of  one  underperforming  program  propagate 
through  the  link  towards  a  neighbor  program  and  affect  it? 

Q5:  Why  is  a  program  that  is  performing  as  expected  not  affected  by 
this  perturbation? 

Q6:  How  does  this  network-centric  approach  facilitate  the 
understanding  of  the  underlying  problems  leading  to  a  cascade  in 
breaches  and  help  the  stakeholders  to  take  appropriate  measures? 


To  address  the  previous  questions  we  employ  the  following  three-phase 
approach: 

Phase  1:  Identify  programs  in  the  MDAP_A  funding  network  that 
underperform  by  analyzing  SAR  files  of  all  programs,  specifically  for  information 
pertaining  to  APB  breaches  and  increases/decreases  in  %PAUC. 

Phase  2:  Study  the  local  reasons  for  the  poor  performance  of  the  programs 
based  on  their  respective  DAES  reports. 

Phase  3:  Study  the  non-local  reasons  for  poor  performance  by  analyzing  the 
SAR  files. 

In  the  next  section,  we  discuss  the  details  of  this  three-phase  approach. 
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III.  Phase  1:  Identify  Programs  in  the  MDAPA 
Funding  Network  That  Exhibit  Poor 
Performance 


We  studied  the  yearly  performance  of  the  MDAP_A  funding  network  using 
SAR  files.  Table  1  shows  the  APB  breaches  and  %PAUC  during  2004-2010  for  the 
nodes  in  the  MDAP_A  network.  Programs  initiated  after  2004  have  data  from  their 
respective  start  date. 


Table  1.  SAR  Summary  of  the  MDAP  A  Funding  Network  for  2004-2010 


MDAP  A 


Schedule 


Performance 


APB 

Breach 

RDT&E  Procurement  PAUC 


2004 


None 


None 


None 


None 


None  (-9.98%) 


2005 


None 


None 


None 


None 


None 


2007 


None 


None 


None 


None 


None 


None 


None 


MDAP  B 


None  (-11.65%) 


None  (-6.14%) 


None  (-1 .24%) 


None  (3.14%) 


None  (3.82%) 


2004 


None 


None 


None 


None 


None 

None  (3.85%) 
None  (3.85%) 
None  (7.69%) 
None  (-26.92%) 
None  (-19.23%) 
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2007 


None 


None  None 


None  (0.93%) 


2009 

None 

None 

None  (-37.79%) 

2010 

None 

None 

None  (-26.75%) 

MDAP D 

2009 

None 

None 

None 

None 

None  (2.45%) 

2010 

None 

None 

None 

None  (1.05%) 

MDAP E 

2006 

None 

None 

None 

None 

None  (- 

10.685%) 

2007 

None 

None 

None 

None 

None  (-4.81%) 

2009 

None 

None 

None 

None 

None  (-3.98%) 

2010 

None 

None 

None 

None 

None  (-11.24%) 

In  SAR  files,  an  APB  breach  is  defined  as  a  condition  in  which  the  value  of 
the  respective  breach  parameters  (schedule,  performance,  RDT&E,  procurement, 
and  PAUC)  is  in  the  range  of  10-15%,  beyond  which  the  condition  is  defined  as  a 
Nunn-McCurdy  breach.  Table  1  captures  whether  a  program  has  APB  breaches  in  a 
given  year  and  what  is  the  %PAUC  of  that  program.  A  program  may  have  more  than 
one  APB  breach  but  experience  a  decrease  in  %PAUC.  For  example,  in  2006  the 
program  MDAP_A  experienced  schedule,  RDT&E,  and  performance  breaches,  yet 
its  %PAUC  decreased.  Two  possible  reasons  could  account  for  this  fact:  (1)  the 
decrease  in  %PAUC  could  be  due  to  a  lagging  effect  from  the  previous  year;  or/and 
(2)  according  to  the  project  management  triangle  model  (Bethke,  2003),  program 
managers  may  intentionally  choose  biases  towards  better  performance  of  one 
component  of  the  program  by  trading  it  off  with  performance  of  other  components. 


Table  1  indicates  that  in  2004  and  2010  the  MDAP_A,  MDAP_B,  and 
MDAP_C  programs  experienced  frequent  APB  breaches  and  increases  in  %PAUC. 
In  order  to  understand  the  causes  of  the  poor  performance  for  these  three  programs, 
we  identified  the  local  causes  for  all  five  programs  and  then  determined  whether 
interdependency  issues  existed  among  them.  In  other  words,  we  observed  whether 
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any  of  these  poorly  performing  programs  propagated  their  performance  effects  to 
other  programs,  causing  the  other  programs  to  perform  poorly  as  well. 
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IV.  Phase  2:  Investigation  of  Local  Reasons  for 
Poor  Performance 

In  this  section,  we  investigate  the  performance  issues  local  to  individual 
MDAPs  and  also  track  how  effective  “mitigation  forecasting”  is  at  resolving  pertinent 
issues.  We  use  the  DAES  reports  of  individual  programs  to  analyze  their 
performance  from  a  local  perspective.  We  observe  that  the  DAES  reports  capture 
the  performance  issues  of  a  program’s  local  domain.  We  focus  on  four  performances 
issues  recorded  in  the  DAES  reports,  namely,  cost,  schedule,  performance,  and 
funding. 

A.  Understanding  the  Local  Causes  for  MDAP_A  to  Perform 
Poorly 

We  studied  a  total  of  40  DAES  reports  for  MDAP_A  that  were  available  from 
2006-2010.  These  reports  are  published  monthly  each  year,  including  the  election 
year  of  2008,  unlike  the  SAR  which  did  not  report  in  2008.  The  program  status  is 
presented  in  DAES  reports  through  the  following  parameters:  cost,  schedule, 
funding,  performance,  and  life-cycle  sustainment  (see  Figure  3).  We  focus  on  cost, 
schedule,  performance,  and  funding  parameters.  Each  parameter  reflects  both  the 
APB  and  contract  status.  The  status  for  each  month  is  represented  in  one  of  three 
colors  depending  on  the  severity  of  the  pertinent  issue.  Green  reflects  the  normal 
state  of  meeting  all  requirements,  yellow  reflects  resolvable  issues  (Resolvable 
APB/Contract),  and  red  refers  to  a  state  that  could  not  meet  the  requirements 
(Critical  APB/Contract). 
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Program  Name:  MDAP  X 


Pre-Decisional  -  For  Official  Use  Only 


Program  Status 


Date: 


■  Cost 

■  Schedule 

■  Performar 

■  Funding 
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Comment:  Contractor  schedule  after  Critical  Design  Review  (CDR)does  not  support  contract  milestones. 


I  Meets  all  Contracts  APB  Reqmts 


Yellow:  Resolvable  Contracts  APB  Issues] 


Pre-Decisional  -  For  Official  Use  Only 


C  an  not  meet  Contracts  APB  Reqmts 
Chart  1 


Figure  3.  Program  Status  of  MDAP  X 

Note.  This  figure  was  taken  from  a  DAES  report  published  April  12,  2010,  for  MDAP_X. 

In  Figures  4-7,  we  present  a  summary  of  the  program  status  of  the  DAES 
reports  for  MDAP_A  in  graphical  form  for  the  four  parameters.  The  horizontal  axis 
shows  the  current  month  while  the  vertical  axis  represents  the  number  of  months 
required  to  resolve  an  issue  (APB  or  contract)  as  captured  by  the  DAES  reports  of 
the  current  month.  Therefore,  these  figures  illustrate  the  effectiveness  of  the 
forecasting  to  resolve  issues.  For  example,  Figure  3  shows  that  in  April  2007  there 
were  resolvable  cost-related  APB  issues  that  would  take  eight  months  to  resolve,  but 
in  September  2007  these  issues  turned  critical,  which  were  again  forecasted  to  be 
resolved  in  eight  months.  However,  by  November  2007,  these  issues  became 
resolvable  and  required  eight  months  to  be  resolved.  By  June  2008,  these  issues 
were  finally  resolved,  but  a  new  resolvable  issue  emerged  that  was  forecasted  to  be 
eliminated  in  eight  months. 
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Figure  4.  Summary  of  Cost-Related  Issues  for  MDAPA 

From  2007-2010 
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Figure  5.  Summary  of  Schedule-Related  Issues  for  MDAP  A 

From  2007-2010 
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Figure  6.  Summary  of  Performance-Related  Issues  for  MDAPA 

From  2007-2010 
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Figure  7.  Summary  of  Funding-Related  Issues  for  MDAP  A 

From  2007-2010 

We  first  determine  how  effective  APB  and  contract  forecasting  are  to  mitigate 
the  pertinent  problems  by  (a)  recording  the  instances  where  the  forecasting  was 
effective  as  well  as  where  it  was  ineffective,  and  (b)  identifying  the  issues  that 
caused  the  predictions  to  slip.  We  then  analyze  the  issues  for  deeper  understanding 
and  categorization. 

In  Tables  2-4,  we  present  our  analyses  in  tabular  format  for  three 
parameters:  cost,  schedule,  and  funding.  Since  MDAP_A  did  not  have  any 
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performance  issues,  we  focus  on  cost,  schedule,  and  funding  issues.  For  each 
parameter,  we  identify  the  number  of  issues  captured  in  the  respective  tables. 

1.  MDAPA  Cost  Analysis 

Table  2  captures  cost-related  issues  for  the  program. 


Table  2.  MDAP  A  Cost  Analysis  Using  DAES  Reports  From  2006-2010 


Current  Status 

Status  at  the 

Predicted  Month 

Causes 

Month:  April  2007 

Issue:  Contract  -  Yellow 

Mitigation  Forecast:  8  months 

Month:  September  2007 

Status:  Contract  -  Red 

Note:  After  5  months  the 
contract  issue  turns  into 
critical 

Issue  1:  hardware 
building 

Issue  2:  hardware  design 

Issue  3:  loqistics  issue 

Month:  September  2007: 

Issue:  Contract  -  Red 

APB  -  Yellow 

Mitigation  Forecast: 

8  months 

Month:  June  2008 

Status:  Contract  -  Yellow 

APB  -  Green 

Issues  1-3:  resolved 

Issue  4:  Contractor 
unable  to  forecast  cost. 

Month:  June  2008 

Issue:  Contract  -  Yellow 

Mitigation  Forecast:  8  months 

Month:  July  2008 

Status:  Contract  -  Red 

Note:  After  1  month  the 
contract  issue  turns  into 
critical 

Issue  4:  Contractor 
unable  to  forecast  cost. 

Issue  5:  Schedule  delay 
increased  contract  cost. 

Month:  July  2008 

Issue:  Contract  -  Red 

Mitigation  Forecast:  4  months 

Month:  December  2008 

Status:  Contract  -  Red 

Issue  4:  Contractor 
unable  to  forecast  cost. 

Issue  5:  Schedule  delay 
increased  contract  cost. 

Month:  December  2008 

Issue:  Contract  -  Red 

Mitigation  Forecast: 

2  months 

Month:  March  2009 

Status:  Contract  -  Red 

Issue  4:  Contractor 
unable  to  forecast  cost. 

Issue  5:  Schedule  delay 
increased  contract  cost. 

Month:  March  2009 

Issue:  Contract  -  Red 

Mitigation  Forecast:  3  months 

Month:  July  2009 

Status:  Contract  -  Yellow 

Issue  4:  remains 

Issue  5:  remains 

Month:  July  2009 

Issue:  Contract  -  Yellow 

Mitigation  Forecast:  8  months 

Month:  April  2010 

Status:  Contract  -  Yellow 

Issue  4:  remains 

Issue  5:  remains 

Lessons  Learned:  Although  Table  2  suggests  that  there  are  some  instances 
where  the  forecasting  turned  out  to  be  effective,  we  observe  and  focus  on  the 
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instances  where  cost-related  forecasting  was  ineffective.  We  identify  two  local 
issues,  namely  (1)  contractors’  inability  to  forecast  cost;  and  (2)  schedule  delays 
leading  to  increased  contract  cost,  which  appear  to  recur  and  lead  to  increased 
program  costs. 

2.  MDAPA  Schedule  Analysis 

Table  3  captures  schedule-related  issues  for  the  program. 

Table  3.  MDAP  A  Schedule  Analysis  Using  DAES  Reports  From  2006-2010 


Current  Status 

Status  at  the  Predicted 
Month 

Causes 

Month:  June  2007 

Issue:  Contract  -  Yellow 

Mitigation  Forecast:  2  months 

Month:  September  2007 

Status:  Contract  -  Yellow 

Note:  August  2007  report  is 
not  available. 

Issue  1 :  Delav  in  MOU 
sign  with  Australia. 

Month:  September  2007 

Issue:  Contract  -  Yellow 

Mitigation  Forecast:  1  month 

Month:  October  2007 

Status:  Contract  -  Green 

Issue  1 :  remains 

Issue  2:  Software  testinq, 
delivery,  and  other 
waveform  issues 

Month:  October  2007 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  1  month 

Month:  November  2008 

Status:  APB  -  Green 

Issue  1 :  resolved 

Issue  2:  resolved 

Month:  March  2008 

Issue:  APB  -  Yellow 

Contract  -  Yellow 
Mitigation  Forecast:  8  months 

Month:  November  2008 

Status:  APB  -  Red 

Contract  -  Red 

Issue  3:  Hardware 
testing  and  performance 
failure 

Issue  4:  Execution  delav 
in  contractor’s  schedule  & 
lack  in  funding 

Month:  November  2008 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  Forecast:  2  months 

Month:  December  2008 

Status:  APB  -  Red 

Contract  -  Red 

Issue  3:  Hardware 
testing  and  performance 
failure 

Issue  4:  Execution  delay 
in  contractor’s  schedule  & 
lack  in  funding 

Month:  December  2008 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  Forecast:  2  months 

Month:  February  2009 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  Forecast: 

2  months 

Issue  3:  Hardware 
testing  and  performance 
failure 

Issue  4:  Execution  delav 
in  contractor’s  schedule  & 
lack  in  funding 
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Month:  February  2009 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  Forecast:  4  months 

Month:  June  2009 

Status:  APB  -  Red 

Contract  -  Red 

Issue  4:  Execution  delay 
in  contractor’s  schedule  & 
lack  in  funding 

Issue  3:  Hardware 
testing  and  performance 
failure 

Month:  June  2009 

Issue:  APB  -  Red 

Contract  -  Yellow 
Mitigation  Forecast: 

APB:  4  months 

Contract:  8  months 

Month:  October  2009 

Status:  APB  -  Green 

Contract  -  Yellow 

Month:  October  2009 

Issue:  Contract  -  Yellow 
Mitigation  Forecast: 

Contract:  5  months 

Month:  March  2010 

Status:  APB  -  Red 

Contract  -  Yellow 

Issue  4:  Execution  delay 
in  contractor’s  schedule  & 
lack  in  funding 

Issue  3:  Hardware 
testing  and  performance 
failure 

Lessons  Learned:  Although  there  are  some  instances  for  which  the 
forecasting  turned  out  to  be  effective,  we  observe  and  focus  on  the  instances  where 
schedule-related  forecasting  was  ineffective.  We  identify  two  local  issues,  namely  (1) 
hardware  testing  and  performance  failure  and  (2)  execution  delay  and  lack  of 
funding,  which  appear  to  recur  and  lead  the  program  towards  schedule  delay. 

3.  MDAP  A  Funding  Analysis 

Table  4  captures  funding-related  issues  for  the  program. 
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Table  4.  MDAP_A  Funding  Analysis  Using  DAES  Reports  From  2006-2010 


Current  Status 

Status  at  the  Predicted 
Month 

Causes 

Month:  April  2007 

Issue:  APB  -  Yellow 

Mitigation  Forecast: 

Contract:  5  months 

Month:  September  2007 

Status:  APB  -  Yellow 

Issue  1 :  WPN  Fund  cut 

Month:  September  2007 

Issue:  APB  -  Yellow 

Mitigation  Forecast: 

Contract:  1  month 

Month:  October  2007 

Status:  APB  -  Yellow 

Issue  1 :  WPN  Fund  cut 

Month:  October  2008 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  Forecast: 

APB:  4  months 

Contract:  2  months 

Month:  December  2008 

Status:  APB  -  Red 

Contract  -  Red 

Issue  1 :  WPN  Fund  cut 

Month:  December  2008 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  Forecast: 

APB:  4  months 

Contract:  2  months 

Month:  February  2009 

Status:  APB  -  Red 

Contract  -  Red 

Issue  1 :  WPN  Fund  cut 

Month:  February  2009 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  Forecast: 

APB:  1  month 

Contract:  1  month 

Month:  March  2009 

Status:  APB  -  Green 

Contract  -  Green 

Month:  April  2009 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  Forecast: 

By  the  current  month 

Month:  May  2009 

Status:  APB  -  Red 

Contract  -  Red 

Issue  1 :  WPN  Fund  cut 

Month:  May  2009 

Issue:  APB  -  Red 

Contract  -  Red 

Mitigation  Forecast: 

4  months 

Month:  September  2009 

Status:  APB  -  Green 

Contract  -  Green 

Lessons  Learned:  Although  there  are  some  instances  for  which  the 
forecasting  turned  out  to  be  effective,  we  observe  and  focus  on  the  instances  where 
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funding-related  forecasting  was  ineffective.  We  identify  one  local  issue,  namely  the 
Weapons  Procurement  Cut,  that  appears  to  recur  and  lead  the  program  to 
experience  funding-related  problems  (for  example,  lack  of  funding  caused  a 
schedule  delay,  as  captured  in  the  MDAP_A  schedule  analyses). 

Based  on  these  lessons  from  the  cost,  schedule,  and  funding  analyses  of 
MDAP_A,  we  identify  the  following  observations  that  appear  to  be  responsible  for 
the  APB  cost  and  schedule  breach  of  MDAP_A: 

Observation  1:  The  design  of  MDAP_A  relies  on  cutting-edge  technology.  It 
seems  that  the  contractor  underestimated  or  could  not  accurately  estimate  the 
technical  challenges  and  the  amount  of  funding  required  to  accomplish  the  tasks. 

Observation  2:  MDAP_A  suffered  greatly  due  to  budget  cuts.  The  program 
did  not  receive  the  required  amount  of  funding  from  the  government  (congressional 
committee),  which  delayed  the  schedule  and,  as  a  consequence,  cost  increased. 

B.  Understanding  the  Local  Causes  for  MDAP_B  to  Perform 
Poorly 

In  this  section,  we  describe  the  analysis  of  44  DAES  reports  for  MDAP_B  that 
were  available  from  2006-2010.  We  first  determined  the  effectiveness  of  APB  and 
contract  forecasting  to  mitigate  the  pertinent  problems.  We  did  this  by  recording  the 
instances  when  the  forecasting  was  effective  as  well  as  when  it  was  ineffective.  We 
then  sought  to  identify  and  analyze  the  issues  that  caused  the  predictions  to  slip. 

In  Figures  8-1 1 ,  we  present  a  summary  of  the  DAES  reports  for  MDAP_B  in 
graphical  form  for  the  four  parameters. 
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Figure  8.  Summary  of  Cost-Related  Issues  for  MDAP_B 

From  2008-2010 
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From  2008-2010 
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Figure  10.  Summary  of  Performance-Related  Issues  for  MDAPB 

From  2008-2010 


Figure  11.  Summary  of  Funding-Related  Issues  for  MDAP  B 

From  2008-2010 

We  present  our  analyses  in  Tables  5-7  for  three  parameters:  cost,  schedule, 
and  funding.  Since  MDAP_B  did  not  have  any  performance  issues,  we  focus  on 
cost,  schedule,  and  funding  issues. 

1.  MDAP_B  Cost  Analysis 

Table  5  captures  cost-related  issues  for  the  program. 
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Table  5.  MDAP  B  Cost  Analysis  Using  DAES  Reports  From  2006-2010 


Current  Status 

Status  at  the  Predicted 

Mllonth 

Causes 

Month:  February  2007 

Issue:  APB -Yellow 

Mitigation  Forecast:  3 
months 

Month:  May  2007 

Status:  APB  -  Yellow 

Issue  1 :  require 
procurement  funding 

Month:  May  2007 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  1 
month 

Month:  June  2007 

Status:  APB  -  Yellow 

Issue  1 :  require 
procurement  funding 

Month:  June  2007 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  2 
months 

Month:  August  2007 

Status:  APB  -  Yellow 

Issue  1 :  require 
procurement  funding 
Issue  2:  contractor 
cost  increased 

Month:  August  2007 

Issue:  APB  -  Yellow 

Contract  -  Yellow 
Mitigation  Forecast:  1 
month 

Month:  September  2007 

Status:  APB  -  Yellow 

Contract  -  Yellow 

Issue  1 :  require 
procurement  funding 
Issue  2:  contractor 
cost  increased 

Month:  September  2007 

Issue:  APB  -  Yellow 

Contract  -  Yellow 
Mitigation  Forecast: 

APB:  1  month 

Contractor:  8  months 

Month:  October  2007 

Status:  APB  -  Green 

Contract  -  Yellow 

Issue  1 :  require 
procurement  funding 

Issue  2:  contractor 
cost  increased 

Month:  October  2007 

Issue:  Contract  -  Yellow 

Mitigation  Forecast:  8 
months 

Month:  June  2008 

Status:  Contract  -  Yellow 

Issue  1 :  require 
procurement  funding 
Issue  2:  contractor 
cost  increased 

Month:  June  2008 

Issue:  Contract  -  Yellow 

Mitigation  Forecast:  8 
months 

Month:  February  2009 

Status:  Contract  -  Yellow 

Issue  1 :  require 
procurement  funding 
Issue  2:  contractor 
cost  increased 

Month:  February  2009 

Issue:  Contract  -  Yellow 
Mitigation  Forecast:  8 
months 

Month:  October  2009 

Status:  Contract  -  Yellow 

Issue  1 :  require 
procurement  funding 

Issue  2:  contractor 
cost  increased 

Month:  October  2009 

Issue:  Contract  -  Yellow 

Mitigation  Forecast:  8 
months 

Month:  April  2010 

Status:  Contract  -  Yellow 

Note:  No  data  available  beyond  April 
2010 

Issue  1 :  require 
procurement  funding 
Issue  2:  contractor 
cost  increased 

NPS 


TwSvtantia  PtR  satNTi/S^J 

MIH1IJ1  - — / 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-30- 


Lessons  Learned:  Although  there  are  some  instances  for  which  the 
forecasting  turned  out  to  be  effective,  we  observe  and  focus  on  the  instances  where 
cost-related  forecasting  was  ineffective.  We  identify  two  local  issues,  namely  (1)  lack 
in  procurement  funding  and  (2)  increased  contract  cost,  which  appear  to  recur  and 
lead  the  program  towards  cost  increase. 

2.  MDAPB  Schedule  Analysis 

Table  6  captures  schedule  issues  for  the  program. 

Table  6.  MDAP  B  Schedule  Analysis  Using  DAES  Reports  From  2006-2010 


Forecasting 

Status  at  the  Predicted  Month 

Causes 

Month:  September  2006 

Issue:  APB -Yellow 
Mitigation  Forecast:  6 
months 

Month:  March  2007 

Status:  APB  -  Yellow 

None  identified 

Month:  March  2007 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  2 
months 

Month:  May  2007 

Status:  APB  -  Yellow 

None  identified 

Month:  May  2007 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  1 
month 

Month:  June  2007 

Status:  APB 

None  identified 

Month:  June  2007 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  2 
months 

Month:  August  2007 

Status:  APB  -  Yellow 

None  identified 

Month:  August  2007 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  1 
month 

Month:  September  2007 

Status:  APB  -  Yellow 

None  identified 

Month:  September  2007 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  2 
months 

Month:  November  2007 

Status:  APB  -  Green 

None  identified 
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Month:  November  2010 
Issue:  APB  -  Yellow 
Mitigation  Forecast:  5 
months 


Month:  April  2010 
Status:  APB  -  Red 

Note:  No  data  available  beyond  April 
2010 


Issue  2:  Phase  1 
Milestone  C  decision 
date  postponement 
and  potential  to  move 
right  beyond 
threshold  date. 

An  MS  C  Threshold 
Breach  causes  the 
Phase  1  (External 
Program)  to  be  red 
for  +3  months 


Lessons  Learned:  Although  there  are  some  instances  for  which  the 
forecasting  turned  out  to  be  effective,  we  observe  and  focus  on  the  instances  where 
schedule-related  forecasting  was  ineffective.  The  DAES  reports  did  not  however 
capture  the  reasons  for  MDAP_B’s  schedule  delay. 

3.  MDAPB  Funding  Analysis 

Table  7  captures  funding-related  issues  for  the  program. 

Table  7.  MDAP  B  Funding  Analysis  Using  DAES  Reports  From  2006-2010 


Forecasting 

Status  at  the  Predicted  Month 

Causes 

Month:  February  2007 

Issue:  APB  -  Yellow 

Contract  -  Yellow 
Mitigation  Forecast:  3 
months 

Month:  May  2007 

Status:  APB  -  Yellow 

Contract  -  Yellow 

Issue  1 :  require 
procurement  funding 

Month:  May  2007 

Issue:  APB -Yellow 

Contract  -  Yellow 

Mitigation  Forecast:  1 
month 

Month:  June  2007 

Status:  APB  -  Yellow 

Contract  -  Yellow 

Issue  1 :  require 
procurement  funding 

Month:  June  2007 

Issue:  APB -Yellow 

Contract  -  Yellow 

Mitigation  Forecast:  2 
months 

Month:  August  2007 

Status:  APB  -  Yellow 

Contract  -  Yellow 

Issue  1 :  require 
procurement  funding 

Month:  August  2007 

Issue:  APB  -  Yellow 

Contract  -  Yellow 

Mitigation  Forecast:  1 
month 

Month:  September  2007 

Status:  APB  -  Green 

Contract  -  Green 

None  identified 
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Month:  March  2008 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  4 
months 

Month:  July  2008 

Status:  APB  -  Yellow 

Issue  1 :  require 
procurement  funding 

Month:  July  2008 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  3 
months 

Month:  October  2008 

Status:  APB  -  Yellow 

Issue  1 :  require 
procurement  funding 

Month:  October  2008 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  3 
months 

Month:  January  2009 

Status:  APB  -  Yellow 

Issue  1 :  require 
procurement  funding 

Month:  January  2009 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  current 
month 

Month:  February  2009 

Status:  APB  -  Yellow 

Issue  1 :  require 
procurement  funding 

Month:  February  2009 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  current 
month 

Month:  March  2009 

Status:  APB  -  Green 

None  identified 

Month:  June  2009 

Issue:  APB  -  Yellow 

Mitigation  Forecast:  5 
months 

Month:  November  2009 

Status:  APB  -  Red 

Issue  2:  R&D 
shortfall  driven  by 
overall  technical  and 
schedule  issues 

Issue  3:  Hardware 
testing  issue  to 
increase  program 
cost 

Month:  November  2009 

Issue:  APB  -  Red 

Mitigation  Forecast:  8 
months 

Month:  April  2010 

Status:  APB  -  Red 

Note:  No  data  available  beyond  April 
2010 

Issue  2:  FY  12-15 

R&D  shortfall  driven 
by  overall  technical 
and  schedule  issues 

Issue  3:  Hardware 
testing  issue  to 
increase  program 
cost 

Lessons  Learned:  Although  there  are  some  instances  for  which  the 
forecasting  turned  out  to  be  effective,  we  observe  and  focus  on  the  instances  where 
funding-related  forecasting  was  ineffective.  We  identify  three  local  issues,  namely  (1) 
lack  of  required  procurement  funding,  (2)  an  R&D  shortfall  driven  by  overall  technical 
and  schedule  issues,  and  (3)  a  hardware  testing  issue  that  increased  program  cost. 
These  issues  appear  to  recur  and  lead  the  program  towards  experiencing  funding- 
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related  problems  (for  example,  a  cost  increase  as  captured  in  the  MDAP_B  cost 
analyses). 

Based  on  these  lessons  learned  from  the  cost,  schedule,  and  funding 
analyses  of  MDAP_B,  we  make  the  following  observations  about  what  is  responsible 
for  the  APB  cost  and  schedule  breach  of  MDAP_B: 

Observation  3:  A  lack  in  procurement  funding  is  the  most  beleaguering  issue 
for  MDAP_B’s  observed  cost  and  funding  problems. 

Observation  4:  The  above  DAES  report-based  analyses  presented  in  Tables 
5-7  ,  however,  do  not  provide  any  clue  as  to  why  the  shortfall  in  funding.  This 
underscores  the  importance  of  looking  beyond  the  local  view  of  a  program  and  of 
searching  for  non-local  causes  that  could  have  contributed  to  the  degradation  in 
performance.  This  motivates  us  to  investigate  the  interdependent  region  between 
MDAP_A  and  MDAP_B  to  identify  possible  cascading  effects. 

C.  Understanding  Local  Issues  of  MDAP_C 

We  studied  a  total  of  46  DAES  reports  for  MDAP_C  that  were  available  from 
2006-2010.  In  this  section,  we  evaluate  the  effectiveness  of  APB  and  contract 
forecasting  to  mitigate  the  pertinent  problems.  To  do  this,  we  recorded  the  instances 
when  the  forecasting  was  effective  as  well  as  when  it  was  ineffective.  We  then 
identified  and  analyzed  the  issues  that  caused  the  predictions  to  slip. 

We  present  our  analyses  in  Tables  8-10  for  three  parameters:  cost,  schedule, 
and  funding.  Since  MDAP_C  did  not  have  any  performance  issues,  we  focus  on  the 
cost,  schedule,  and  funding  issues. 

1.  MDAP_C  Cost  Analysis 

Table  8  captures  cost-related  issues  for  the  program. 


NAVAL  POSTGRADUATE  SCHOOL 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 


-34- 


Table  8.  MDAPC  Cost  Analysis  Using  DAES  Reports  From  2006-2010 


Current  Status 

Status  at  the  Predicted 
Month 

Causes 

Month:  March  2007 

Issue:  APB  -  Red 

Mitigation  Forecast:  current 
month 

Month:  April  2007 

Status:  APB  -  Red 

Issue  1:  FY08  PB  caused  an 
APB  cost  breach  in  the 

PAUC. 

Month:  April  2007 

Issue:  APB  -  Red 

Mitigation  Forecast:  8  months 

Month:  December  2007 

Status:  APB  -  Green 

Note:  Issue  1  resolved  due  to 
approved  baseline. 

Lessons  Learned:  We  did  not  observe  any  noticeable  cost-related  issue. 

2.  MDAP  C  Schedule  Analysis 

Table  9  captures  schedule  issues  for  the  program. 

Table  9.  MDAP_C  Schedule  Analysis  Using  DAES  Reports  From  2006-2010 


Current  Status 

Status  at  the  Predicted 
Month 

Causes 

Month:  March  2007 

Issue:  APB -Yellow 

Mitigation  Forecast:  current 
month 

Month:  April  2007 

Status:  APB  -  Green 

None  identified 

Month:  July  2007 

Issue:  Contract  -  Yellow 
Mitigation  Forecast:  8  months 

Month:  March  2008 

Status:  Contract  -  Yellow 

Issue  1:  Contract  Schedule 
is  yellow  due  to  IDIQ 
Production  and  Production 
facility  transition  contract 
award,  expected  1st  Qtr  & 

2nd  Qtr  FY09,  respectively. 

Month:  October  2007 

Issue:  APB -Yellow 

Mitigation  Forecast:  8  months 

Month:  December  2007 

Status:  APB  -  Red 

Issue  2:  The  slip  in  the 
release  of  the  MOT&E-2 
report  could  potentially  delay 
the  IOC-2/3  decision  and 
cause  an  APB  breach. 

Month:  December  2007 

Issue:  APB  -  Red 

Mitigation  Forecast:  2  months 

Month:  February  2008 

Status:  APB  -  Red 

Issue  2:  APB  schedule  is  red 
because  the  slip  in  the 
release  of  the  MOT&E-2 
report  delayed  the  IOC-2/3 
decision  past  the  31  Dec  07 
APB  threshold,  which 
resulted  in  an  APB  breach. 

Month:  February  2008 

Issue:  APB  -  Red 

Month:  June  2008 

Status:  APB  -  Red 

Issue  2:  APB  schedule  is  red 
because  the  slip  in  the 
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Mitigation  Forecast:  4  months 

release  of  the  MOT&E-2 
report  delayed  the  IOC-2/3 
decision  past  the  31  Dec  07 
APB  threshold,  which 
resulted  in  an  APB  breach. 

Month:  March  2008 

Issue:  Contract  -  Yellow 

Mitigation  Forecast:  8  months 

Month:  November  2008 

Status:  Contract  -  Green 

None  identified 

Month:  June  2008 

Issue:  APB  -  Red 

Mitigation  Forecast:  1  month 

Month:  July  2008 

Status:  APB  -  Red 

Issue  2: 

IOC  2/3  current  estimate  of 

30  Jun  08  has  slipped  to  Oct 

08  due  to  Joint  Staff  critical 
comments. 

Month:  July  2008 

Issue:  APB  -  Red 

Mitigation  Forecast:  3  months 

Month:  November  2008 

Status:  APB  -  Green 

None  identified 

Lessons  Learned:  The  only  issue  that  affected  the  schedule  is  the  following 
Completion  of  the  Initial  Operational  Capability  (IOC)  2/3  milestone  was  delayed 
because  of  changes  in  reporting  procedures  and  evaluation  criteria  at  the 
operational  test  agency  while  this  milestone  was  under  evaluation.  As  a  result,  the 
final  multi-service  Operational  Test  and  Evaluation  or  MOT&E-2  report  from 
AFOTEC  was  delayed. 

3.  MDAP_C  Funding  Analysis 

Table  10  captures  funding-related  issues  for  the  program. 
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Table  10.  MDAP_C  Funding  Analysis  Using  DAES  Reports  From  2006-2010 


Current  Status 

Status  at  the  Predicted 
Month 

Causes 

Month:  March  2008 

Issue:  Contract  -  Yellow 

Mitigation  Forecast:  8 
months 

Month:  December  2008 

Status:  Contract  -  Green 

None  identified 

Month:  January  2009 

Issue:  Contract  -  Yellow 

Mitigation  Forecast:  8 
months 

Month:  October  2009 

Status:  Contract  -  Yellow 

None  identified 

Month:  October  2009 

Issue:  Contract  -  Yellow 

Mitigation  Forecast:  8 
months 

Month:  April  2009 

Status:  Contract  -  Yellow 

None  identified 

Lessons  Learned:  We  cannot  find  any  significant  information  on  the  funding 
issues  from  the  DAES  reports  on  MDAP_C. 

D.  Understanding  Local  Issues  of  MDAP_D 

We  studied  a  total  of  24  DAES  reports  for  MDAP_D  available  from  2008- 
2010.  We  first  determined  the  effectiveness  of  APB  and  contract  forecasting  to 
mitigate  the  pertinent  problems.  To  do  this,  we  recorded  the  instances  when  the 
forecasting  was  effective  as  well  as  when  it  was  ineffective.  We  then  identified  and 
analyzed  the  issues  that  caused  the  predictions  to  slip. 

We  present  our  analyses  in  Tables  1 1  and  12  for  two  parameters:  cost  and 
schedule.  Since  MDAP_D  did  not  have  any  funding-  or  performance-related  issues, 
we  focus  on  the  cost  and  schedule  issues. 

MDAP_D  Cost  Analysis 

1 1  captures  cost-related  issues  for  the  program. 
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Table  11.  MDAPD  Cost  Analysis  Using  DAES  Reports  From  2008-2010 


Current  Status 

Status  at  the  Predicted 
Month 

Causes 

Month:  May  2009 

Issue:  Contract  -  Yellow 
Mitigation  Forecast: 

8  months 

Month:  November  2009 

Status:  Contract  -  Yellow 

None  identified 

Month:  November  2009 

Issue:  Contract  -  Yellow 

Mitigation  Forecast: 

2  months 

Month:  January  201 0 

Status:  Contract  -  Green 

None  identified 

Month:  January  2010 

Issue:  Contract  -  Yellow 

Mitigation  Forecast: 

4  months 

Month:  April  2010 

Status:  Contract  -  Yellow 

None  identified 

Lessons  Learned:  We  did  not  observe  any  cost-related  issue. 

2.  MDAP  D  Schedule  Analysis 

Table  12  captures  schedule  issues  for  the  program. 
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Table  12.  MDAP  D  Schedule  Analysis  Using  DAES  Reports  From  2008-2010 


Current  Status 

Status  at  the  Predicted 
Month 

Causes 

Month:  April  2009 

Issue:  Contract  -  Yellow 
Mitigation  Forecast: 

1  month 

Month:  May  2009 

Status:  Contract  -  Yellow 

Issue  1:  The  Critical  Design 
Review  (CDR)  date  is 
expected  to  move  to  Sep  09 
in  order  to  accommodate 

Joint  Industry/Government 
Tiger  Team 

recommendations  to  reduce 
Information  Assurance 
certification  issues. 

Issue  2:  The  contractor 
schedule  may  not  support 
testing  required  to  satisfy  exit 
criteria  to  meet  the  MSC 
objective  date  of  Nov  2011. 

Month:  May  2009 

Issue:  Contract  -  Yellow 

Mitigation  Forecast: 

5  months 

Month:  October  2009 

Status:  Contract  -  Yellow 

APB  -  Yellow 

Issue  1:  remains 

Issue  2:  remains 

Month:  October  2009 

Issue:  APB  -  Yellow 

Contract  -  Yellow 
Mitigation  Forecast: 

3  months 

Month:  January  201 0 

Status:  Contract  -  Yellow 

Issue  2:  remains 

Issue  3:  Potential  impact  to 
program  schedule  as  a  result 
of  concurrent  software  and 
hardware  development 
interdependencies. 

Month:  January  2010 

Issue:  Contract  -  Yellow 
Mitigation  Forecast: 

4  months 

Month:  April  2010 

Status:  Contract  -  Yellow 

Issue  2:  resolved. 

Issue  3:  Potential  impact  to 
program  schedule  as  a  result 
of  concurrent  software  and 
hardware  development 
interdependencies. 

Lessons  Learned:  We  identify  that  concurrent  waveform  and  hardware 
development  interdependencies  may  have  affected  the  program  schedule. 

E.  Understanding  Local  Issues  of  MDAP_E 

We  studied  a  total  of  41  DAES  reports  for  MDAP_E  that  were  available  from 
2007-2010.  We  first  determined  the  effectiveness  of  APB  and  contract  forecasting  to 
mitigate  the  pertinent  problems.  To  do  this,  we  recorded  the  instances  when  the 
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forecasting  was  effective  as  well  as  when  it  was  ineffective.  We  then  identified  and 
analyzed  the  issues  that  caused  the  predictions  to  slip. 

We  cannot,  however,  find  significant  degradation  in  terms  of  cost-,  schedule-, 
funding-,  and  performance-related  issues  for  MDAP_E. 

The  above  DAES  report-based  analyses  presented  in  Tables  8-12  indicate 
that  MDAP_C,  MDAP_D,  and  MDAP_E  may  not  have  affected  the  performance  of 
MDAP_A  and  it  appears  that  there  could  be  potential  interdependency  between 
MDAP_A  and  MDAP_B.  We,  therefore,  compare  and  investigate  the  issues  related 
to  these  two  programs  in  an  effort  to  better  understand  the  possible 
interdependencies. 

In  Table  13,  we  provide  a  summary  of  findings  revealed  from  the  study  of 
DAES  reports  for  MDAP_A  and  MDAP_B,  in  an  effort  to  understand  the  non-local 
issues. 


Table  13.  MDAP  A  and  MDAP  B  Local  Issue  Summary  for  2006-2010 


MDAP_A  Issues 

MDAP_B  Issues 

-  Contractor’s  inability  to  forecast  cost 

-  Schedule  delay  increased  contract 
cost 

-  Hardware  testing  and  performance 
failure 

-  Execution  delay  and  lack  of  funding 

-  Lack  in  procurement  funding 

-  Increased  contract  cost 

-  Required  procurement  funding 

-  R&D  shortfall  driven  by  overall  technical 
and  schedule  issues 

-  Hardware  testing  issue  to  increase 
program  cost 

Table  13  indicates  that  although  the  contractor’s  ineffective  forecasting  and 
schedule  delay  (due  to  hardware  and  design  issues)  led  MDAP_A  to  incur  a  cost 
overrun,  the  lack  in  procurement  funding  appears  to  be  the  plaguing  issue  for  the 
increase  in  cost  of  MDAP_B.  Based  on  this  observation,  we  propound  the  following 
hypothesis:  The  cost  increase  of  MDAP_A  in  2009  could  have  caused  the 
procurement  funding  shortfall  for  MDAP_B  in  2010,  which,  in  effect,  increased  the 
cost  of  MDAP_B  (as  the  DAES  reports  on  MDAP_B  suggest). 
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This  hypothesis  indicates  that  the  cost  increase  of  MDAP_A  (in  2009)  is  due 
to  non-local  reasons.  DAES  reports  do  not  provide  enough  insight  about 
interdependency  regions  and,  hence,  we  could  not  capture  the  non-local  causes 
from  these  reports.  However,  we  observe  that  the  SAR  files  to  some  extent  offer  a 
better  perspective  to  understand  the  interdependency  region.  In  the  following 
section,  we  provide  a  comparative  analysis  of  the  funding  phenomena  for  these  two 
programs  in  an  effort  to  discover  interdependency  between  the  cost  increase 
problem  of  MDAP_A  and  funding  shortfall  of  MDAP_B. 
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V.  Phase  3:  Study  of  the  Non-Local  Reasons  for 
Poor  Performance  by  Analyzing  the  SAR 


To  verify  the  hypothesis  of  interdependency,  we  created  Tables  14  and  15, 
which  contain  the  funding  summary  (based  on  base  year  dollar)  from  the  SAR  files 
of  MDAP_A  and  MDAP_B  for  the  period  2004-2010.  Our  study  indicates  that 
comparative  analyses  of  SAR  files  for  the  programs  in  MDAP_A’s  funding  network 
provide  insight  about  the  joint  space  and,  hence,  are  useful  for  us  to  identify  non¬ 
local  issues.  SAR  captures  the  yearly  APB  breach  status,  %PAUC,  cost,  and  funding 
data;  hence,  it  is  suitable  for  quantitative  analyses. 

Table  14.  MDAP_A  SAR  Funding  Summary  ($BY)  for  the  Period  2004-2010 


MDAP_A 

Baseline 

Current 

%PAUC 

Current  Year 

Received 

Funding 

Delta 

Quantity 

Quantity 

Required  Funding 

(x) 

(y) 

(y-x) 

2004 

6 

6 

-9.98 

221.1 

2005 

6 

6 

-11.65 

598.5 

579.8 

-18.7 

2006 

6 

6 

-6.14 

1012.1 

997.3 

-14.8 

2007 

6 

6 

-1.24 

1588.4 

1574.6 

-13.8 

2009 

6 

6 

3.14 

3163.2 

3006.3 

-156.9 

2010 

6 

6 

3.82 

3750.7 

3813.2 

62.5 

Table  15.  MDAP_B  SAR  Funding  Summary  ($BY)  for  the  Period  2004-2010 


MDAP_B 

Baseline 

Current 

%PAUC 

Current  Year 

Received 

Funding 

Delta 

Quantity 

Quantity 

Required  Funding 
(x) 

(y) 

(y-x) 

2004 

329574 

329574 

0 

44.2 

44.2 

0 

2005 

329574 

328514 

3.85 

137.2 

135.5 

-1.7 

2006 

329574 

328514 

3.85 

255.5 

250.3 

-5.2 

2007 

329574 

95961 

7.69 

350.5 

348.1 

-2.4 

2009 

329574 

215961 

-26.92 

644.1 

593.2 

-50.9 

2010 

329574 

221978 

-19.23 

751.6 

711.1 

-40.5 

NPS 
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In  Tables  14  and  15,  we  focus  on  the  parameter  “delta,”  which  captures  the 
difference  in  the  amount  of  required  and  received  funding  for  the  respective  year. 

For  MDAP_A,  we  notice  that  from  2009-2010  the  %PAUC  increased  while  delta 
turned  out  to  be  positive.  On  the  other  hand,  for  MDAP_B  from  2009-2010,  delta 
retained  a  large  negative  value  even  though,  given  the  trends  over  the  years,  the 
increase  in  quantity  (-4000  units)  was  not  large  enough  to  justify  this  increase.  Both 
the  DAES  and  SAR  files  of  MDAP_B  do  not  provide  reasons  for  the  large  negative 
value  of  delta  in  2009  and  2010.  We  suspect  that  the  cost  overrun  of  MDAP_A  from 
2009  onward  might  have  affected  MDAP_B  in  2010  through  a  procurement  funding 
shortfall.  This  observation,  even  if  it  may  not  be  conclusive,  is  suggestive  of 
cascading  effects  between  neighboring  MDAPs.  We  believe  that  a  thorough  study  of 
the  entire  set  of  MDAPs  may  enable  us  to  find  more  interesting  interdependencies 
and  would  be  able  to  predict  the  flow  of  the  cascading  effects. 
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VI.  Observations  From  the  Performance  Reports- 
Based  Analyses 

We  studied  the  DAES  and  SAR  files  of  MDAP_A  and  MDAP_B  available  from 
2006-2010  in  an  effort  to  identify  a  cascading  effect  in  the  MDAP_A  funding 
network.  We  tried  to  understand  the  local  as  well  as  non-local  issues  that  led  the 
programs  towards  breach  condition.  The  following  are  the  summary  of  observations 
from  this  process. 

Observation  1:  The  design  of  MDAPs  relies  on  cutting-edge  technology.  It 
appears  that  the  contractor  either  underestimates  or  cannot  accurately  estimate  the 
technical  challenges  and  the  amount  of  funding  required  to  accomplish  the  tasks. 

Observation  2:  Programs  are  affected  greatly  by  budget  cuts.  Sometimes  a 
program  does  not  receive  the  required  amount  of  funding  from  the  government 
(congressional  committee),  which  delays  the  schedule,  and,  as  a  consequence,  cost 
increases. 

Observation  3:  Lack  in  procurement  funding  is  another  cause  that  leads  to 
cost  and  funding  problems. 

Observation  4:  Analyses  of  the  local  issues  and  the  fact  that  some  of  the 
issues  are  recurrent  indicate  that  either  the  root  cause  of  the  problem  is  not  captured 
in  the  DAES  documents  or  that  the  cause  is  exogenous  to  the  program  boundary. 

Observation  5:  Analyses  of  SAR  files,  on  the  other  hand,  offer  some  insight 
about  the  interdependency  of  the  programs. 

Observation  6:  The  observed  instance  of  a  possible  cascading  effect  in  the 
MDAP_A  network  motivates  us  to  design  an  automated  scheme  that  would  be  able 
to  identify  and  predict  the  likelihood  of  cascading  effects. 
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VII.  Progress  Towards  a  Decision-Theoretic  Model 
for  the  MDAP  Network 

In  this  section,  we  describe  our  initial  work  in  determining  whether  a  decision- 
theoretic  model  for  probabilistic  analysis  of  decisions  is  feasible  given  the  data  we 
have  available  about  the  MDAP  case  study  network.  Our  goal  is  do  a  small-scale 
feasibility  study  in  this  project  and  leave  the  implementation  of  the  model  to  future 
work  where  we  expect  to  have  access  to  larger  amounts  of  data,  which  are  crucial 
for  accurate  modeling  and  analysis. 

A  Markov  Decision  Process  (MDP;  Bertsekas,  1987)  is  a  probabilistic  model 
for  decision-making  and  planning.  It  uses  dynamic  programming  to  decide  on  the 
optimal  actions  (in  this  case,  cut  funding  by  50%  or  delay  schedule  by  six  months) 
that  yields  the  highest  expected  utility  (for  example,  no  PAUC  growth  or  no  APB 
breaches).  MDPs  capture  the  essence  of  sequential  processes  and  are  used  to 
compute  decision  policies  that  lead  to  the  best  long-term  performance  for  the  entire 
network. 

In  theory,  MDPs  implement  two  forms  of  hedging  that  can  allow  managers  to 
(1 )  test  their  decisions  to  avoid  the  possibility  of  failure  and  (2)  choose  actions  that 
ensure  higher  overall  expected  reward.  These  hedging  strategies  alter  expectations 
about  future  problem  occurrences  in  a  manner  that  allows  managers  to  shift 
behaviors  to  improve  performance. 

In  our  approach,  MDAPs  are  considered  as  individual  agents  that  are  part  of 
a  cooperative  multiagent  system,  and  decision-making  in  an  MDAP  network  is 
viewed  as  a  multiagent  sequential  decision  problem  because  the  utility  gained  by 
each  agent  depends  on  a  sequence  of  actions  over  time.  Our  goal  is  to  determine 
the  behavior  of  the  agents  that  best  balances  the  risks  and  rewards  while  acting  in 
an  uncertain  environment  with  stochastic  actions. 
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Each  MDAP  makes  its  individual  decisions  in  an  environment  where  the  state 
space  is  not  fully  observable,  meaning  that  the  nodes  in  the  network  (the  programs) 
do  not  exactly  know  which  state  they  are  in  at  any  particular  instant  because  they  do 
not  have  complete  information  about  their  neighbors.  With  the  partial  state 
information,  the  individual  agents  aim  to  optimize  the  joint  reward  function.  This 
class  of  problems  is  modeled  as  decentralized  partially  observable  MDP  (DEC- 
POMDP)  in  the  literature  (Bernstein  et  al. ,  2002)  where  at  each  step  when  an  agent 
takes  an  action,  a  state  transition  occurs  and  the  agent  receives  a  local  observation. 
Following  this,  the  environment  generates  a  global  reward  that  depends  on  the  set  of 
actions  taken  by  all  the  agents.  The  complexity,  however,  of  this  decentralized 
control  model  is  NEXP-hard  (Bernstein  et  al.,  2002)  and,  hence,  it  is  computationally 
intractable.  In  our  previous  work  (Cheng,  Raja,  &  Lesser,  2012),  we  make  the  DEC- 
POMDP  problem  for  tornado  tracking  tractable  by  approximating  the  DEC-POMDP 
with  a  stochastic  DEC-MDP1  model  and  using  a  factored  reward  function  to  define  a 
Nash  Equilibrium  instead  of  the  global  reward  function.  A  necessary  condition  for 
stable  equilibrium  among  agents  in  a  multiagent  system  is  that  each  agent  plays  a 
best-response  to  the  strategy  of  every  other  agent:  This  is  called  a  Nash  Equilibrium. 
We  apply  this  technique  to  the  MDAP  domain.  We  define  the  reward  function  of  this 
model  to  be  composed  of  two  different  components:  local  reward  function  and  global 
reward  function.  The  local  reward  functions  are  dependent  only  on  the  individual 
agents’  actions,  while  the  global  reward  function  depends  on  the  action  of  all  agents. 
Stochastic  models  are  known  to  cope  with  the  uncertainty  of  observation  and 
perform  better  than  deterministic  policies  in  a  partially  observable  environment. 

The  stochastic  DEC-MDP  model  is  formally  defined  as  a  tuple  <5r<ArT.7l>, 

where  *  = x  s2  x . x£„  is  a  finite  set  of  factored  world  states,  and  where  st  is  the 

state  space  of  agent  /.  Also,  ^  =  A.  xAl  x....xA„  is  a  finite  set  of  joint  actions,  where 
A i  is  the  action  set  for  agent  /. 


1  A  DEC-MDP  is  a  DEC-POMDP  with  joint  full  observability  (Bernstein  et  al.,  2002). 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-48  - 


7: &  xc/i  x  j"  ->7i  is  the  transition  function.  T  ('s  j  s,  a)  is  the  probability  of 
transiting  to  the  next  state  after  a  joint  action  a  -  A  is  taken  by  agents  in  state  s. 

7.  =  /£,  F:2,.. j?„j  is  a  set  of  factored  reward  functions.  F,  :  S  x  A  -> 7.  provides 
agent  /  with  an  individual  reward  r£  c  R^s,  a)  for  taking  action  a  in  state  s. 

A  stochastic  policy  of  an  agent  /  is  denoted  by  IJi(s)  ePD(Aj),  where  PD(Aj),  is 
the  set  of  probability  distributions  over  actions  A-,. 

A.  State  Space 

Feature  1:  Program  ID 
Feature  2:  Current  Year 
Feature  3:  Current  Month 

Feature  4:  Cost  (APB)  Status:  for  nine  months,  starting  from  the  current  month 

Feature  5:  Cost  (Contract)  Status:  for  nine  months,  starting  from  the  current  month 

Feature  6:  Schedule  (APB)  Status:  for  nine  months,  starting  from  the  current  month 

Feature  7:  Schedule  (Contract)  Status:  for  nine  months,  starting  from  the  current 
month 

Feature  8:  Performance  (APB)  Status:  for  nine  months,  starting  from  the  current 
month 

Feature  9:  Performance  (Contract)  Status:  for  nine  months,  starting  from  the  current 
month 

Feature  10:  Funding  (APB)  Status:  for  nine  months,  starting  from  the  current  month 

Feature  1 1:  Funding  (Contract)  Status:  for  nine  months,  starting  from  the  current 
month 

Features  4-1 1  are  represented  by  one  of  three  colored  bubbles  (green, 
yellow,  and  red)  in  the  Program  Status  page  of  the  DAES  report.  Yellow  bubbles 
refer  to  resolvable  issues,  and  red  bubbles  refer  to  critical  issues.  If  there  is  no  issue, 
then  the  feature  is  represented  by  a  green  bubble.  The  number  of  bubbles  starting 
from  the  current  month  indicates  the  number  of  months  during  which  the  issue  is 
sustained.  We  assign  the  green,  yellow,  and  red  bubbles  weights  of  0.0,  0.1 ,  and 
1 .0,  respectively.  Hence,  in  the  feature  value,  the  count  of  yellow  bubbles  will  appear 
at  the  right  side  of  the  decimal  point  and  the  count  for  red  bubbles  will  appear  at  the 
left  side  of  the  decimal  point  of  the  feature  value.  For  example,  consider  the  value  of 
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Feature  4:  Cost  (APB)  =  4.0.  This  value  indicates  that  the  Cost  (APB)  issue  is  critical 
and  that  it  will  continue  to  be  critical  for  next  consecutive  four  months  after  which 
time  it  is  expected  to  be  resolved. 

B.  Action  Space 

We  capture  both  local  and  non-local  actions. 

Local  Action  1  (LAI ):  PM  takes  action  to  resolve  APB  cost  issue. 

Local  Action  2  (LA2):  Contractor  takes  action  to  resolve  contractor  cost  issue. 

Local  Action  3  (LA3):  PM  takes  action  to  resolve  APB  schedule  issue. 

Local  Action  4  (LA4):  Contractor  takes  action  to  resolve  contractor  funding  issue. 

Local  Action  5  (LA5):  PM  takes  action  to  resolve  APB/contractor  funding  issue. 

Local  Action  6  (LA6):  PM  takes  action  to  resolve  APB  performance  issue. 

Local  Action  7  (LA7):  Contractor  takes  action  to  resolve  contractor  performance 

issue. 

Local  Action  8  (LA8):  PM  initiates  inter-governmental  dialogue  to  resolve  the 

pertinent  issue. 

Non-Local  Action  (NLA):  Coordinate  with  program  /.  (/  refers  to  a  neighbor  program) 

C.  Transition  Probabilities 

When  we  extend  this  initial  model  to  real  data  as  part  of  out  future  work,  the 
transition  probability  function  will  be  computed  empirically  based  on  the  past 
performance  breaches  of  programs  in  the  network.  Accurate  definition  of  this 
function  will  require  large  amounts  of  data  about  the  performance  and  breaches  of  a 
large  number  of  MDAP  programs  over  long  periods  (e.g.,  monthly  performance 
information  for  five  or  more  years  for  50  or  more  MDAPs  would  be  a  good  place  to 
start).  Access  to  data  at  this  scale  will  require  automated  analysis  of  the  data 
sources,  which  is  one  of  our  goals  for  the  next  phase  of  the  project. 

D.  Reward  Function 

The  joint  reward  function  is  composed  of  local  and  global  rewards.  Local 
rewards  are  achieved  both  monthly  and  yearly.  Also  as  part  of  future  work,  we 
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propose  to  calculate  local  reward  value  from  the  Acquisition  Baseline  Program 
section  of  DAES  reports.  The  following  two  parameters  (LR1  and  LR2)  capture  the 
change  in  %PAUC  and  Schedule  on  a  monthly  basis.  We  use  the  following  code  to 
depict  their  changes: 

If  current  %PAUC  <  10%  of  the  APB,  then  status  =  0. 

If  0  <  current  %PAUC  <  10%  of  the  APB,  then  status  =  +. 

If  1 0%  of  APB  <  current  %PAUC  <  1 5%  of  the  APB,  then  status  =  1 . 

If  current  %PAUC  >  15%  of  the  APB,  then  status  =  10  (breach  has  occurred). 
Schedule:  #  of  months  beyond  the  threshold 
LR1 :  PAUC_Monthly  (APB) 

LR2:  Schedule_Monthly  (APB) 

To  calculate  the  local  reward  value  that  is  calculated  yearly,  we  use  the 
following  parameters  captured  from  SAR  files: 

LR3:  APB  Breach  RDT&E  (Values:  0/1) 

LR4:  APB  Breach  Procurement  (Values:  0/1) 

LR5:  APB  Breach  Schedule  (Values:  0/1) 

LR6:  APB  Breach  Performance  (Values:  0/1) 

LR7:  APB  Breach  PAUC  (Values:  0/1) 

LR8:  Nunn-McCurdy  Breach  PAUC  (Values:  0/1) 

LR9:  %PAUC  (amount  that  appears  in  SAR) 
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As  part  of  another  project,  we  are  studying  the  viability  of  defining  the  global 
reward  associated  with  an  MDP  state  using  the  “centrality  measure”  from  network 
theory  (Newman,  2011)  to  capture  the  importance/influence  the  state  has  on 
cascading  consequences  downstream. 
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VIII. 


Understanding  the  Characteristics  of  the 
Existing  Data 


In  this  section  we  describe  the  importance  of  the  DAES  data  set  that 
facilitates  deeper  understanding  about  the  dynamics  of  the  MDAP  network.  We  also 
enumerate  the  issues  related  to  the  quality  of  the  data  as  well  as  its  completeness 
and  availability.  We  believe  that  by  addressing  these  issues,  the  accuracy  of  the 
proposed  decision-theoretic  model  will  be  enhanced. 
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Figure  12.  Completeness  of  MDAP  A  2006-2010  DAES  data.  Green  =  full  report;  yellow 

=  partial  report;  red  =  no  report 

A.  Significance  of  the  Data  Set 

The  available  data  that  we  used  for  in-depth  study  of  the  MDAP_A  funding 
network  offers  significant  insight  into  each  individual  program  as  well  as  the 
programs’  interdependency  relationships.  DAES  reports,  which  are  published 
monthly,  provide  a  granular  view  of  the  local  issues  pertaining  to  the  program  and 
the  mitigation  actions  that  have  been  taken  to  resolve  the  issues.  Analyses  of 
monthly  forecasting  on  the  program  features  help  us  to  identify  the  root  cause  of  the 
program  issues  locally.  Failure  to  identify  local  root  causes  results  in  the  search  for 
non-local  causes  that  originated  through  cascading  effects.  SAR  files,  on  the  other 
hand,  provide  a  quantitative  depiction  of  the  program  status  on  the  basis  of  accrued 
breaches  and  of  increases  in  %PAUC,  cost,  and  funding  figures.  This  resource  helps 
us  for  comparative  quantitative  analyses  and  to  gain  insight  about  cascading  effects. 

B.  Structure  of  the  Data 

We  note  that  none  of  the  performance  reports  directly  capture  the 
interdependent  regions. 

Although  the  PE  documents  (R-docs)  provide  a  set  of  programs  that 
share  a  common  funding  source,  they  do  not  provide  a  comparative 
status  of  the  programs. 

The  DAES  reports  show  the  data  interdependency,  but  do  not  provide 
a  summary  status  of  the  data  neighbors. 

To  determine  the  cascading  effect  between  MDAP_A  and  MDAP_B, 
we  had  to  build  a  “funding  summary”  table  for  both  programs  based  on 
base  year  dollar.  The  existing  SAR  format  provides  only  the  then-year 
funding  summary.  For  comparison  and  analyses,  this  table  should  be 
provided  in  terms  of  base  year  dollars. 
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■  We  observe  that  some  DAES  reports  provide  better  understanding  of 
the  issues  and  mitigation  measures,  while  others  do  not.  There  should 
be  a  uniform  standard  to  prepare  this  document. 

C.  Availability  of  Data 

■  We  observe  that  monthly  DAES  reports  for  the  nodes  in  the  MDAP_A 
funding  network  provide  a  very  small  spectrum  of  useful  data  for 
analyses.  Although  some  programs  report  from  2006,  the  complete 
data  set  for  all  the  members  of  the  MDAP_A  network  is  available  only 
for  2008  and  2009.  For  2007,  only  some  programs  possess  complete 
reports.  Some  DAES  reports  provide  partial  information  (contain  only 
the  risk  summary  page)  and,  hence,  are  not  suitable  for  our  analysis. 

■  So  far,  SAR  seems  to  be  the  only  resource  that  captures  some  aspects 
of  interdependency.  But  the  fact  that  SAR  was  not  published  in  2008 
caused  discontinuity  in  our  analyses. 

■  We  find  that  some  programs  stopped  reporting  after  a  certain  time. 
Therefore,  we  had  no  way  to  learn  the  status  of  the  program,  even  if  it 
performed  poorly.  This  unavailability  problem  appears  to  be  a 
challenge  for  understanding  interdependency  issues. 
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IX.  Challenges  Due  to  Missing  Data 


This  paper  is  based  on  the  data  that  we  received  in  August  2011.  The  funding 
network  for  MDAP_A  based  on  the  data  analysis  is  shown  in  Figure  2.  But  the  recent 
data  that  we  received  in  October  2012  indicates  that  MDAP_A  has  only  two 
neighbors:  MDAP_C  and  MDAP_E  (see  Figure  13).  This  was  because  our  initial 
data  were  based  on  neighborhood  information  extracted  from  PE  reports.  Later  we 
found  that  there  were  inaccuracies  in  the  data  and  had  to  correct  the  MDAP_A 
neighborhood  information.  Based  on  the  updated  network  for  MDAP_A,  we  have 
conducted  a  new  study  and  investigated  the  DAES  reports  for  the  members  of  the 
updated  MDAP_A  funding  network.  In  an  effort  to  create  an  MDAP  model,  we 
extracted  the  state  features  for  these  member  programs.  Moreover,  we  have  done 
manual  data  analyses  to  understand  possible  interdependency  between  the 
programs.  In  the  manual  analyses  described  in  Figure  14,  our  aim  was  to  categorize 
all  observed  issues  that  affected  the  programs  and  also  to  identify  the 
interrelationship  between  the  inter-program  issues  that  could  have  triggered 
cascading  effects.  We  went  through  the  issue  summary  sections  in  the  DAES 
documents,  which  are  authored  by  different  program  managers,  and  identified  the 
phrases  that  were  categorized  to  cost,  schedule,  funding,  and  performance.  For 
instance,  “cost  growth”  and  “unable  to  provide  accurate  cost  forecast”  were  used  to 
describe  cost-related  issues.  We  plan  to  use  this  manually  extracted  categorical 
information  to  train  learning  algorithms  that  will  be  used  to  extract  relevant  data 
automatically  in  our  future  work. 


Figure  13.  Updated  MDAP  A  Funding  Network 
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Figure  14.  Categorization  of  the  Issues  as  Captured  From  the  DAES  Reports 

for  the  Updated  MDAP  A  Funding  Network 

We  now  provide  a  flow  diagram  (Figure  15)  that  shows  a  potential  cascading 
effect  between  MDAP_A  (cost  growth  in  2007)  and  MDAP_C  (funding  shortfall  in 
2008).  This  observation,  however,  is  neither  conclusive  nor  complete  unless  we 
investigate  the  first-order  funding  partners  of  the  program  MDAP_C.  Currently,  we 
are  extending  our  study  over  the  first-order  neighbors  of  the  members  of  the 
MDAP_A  funding  network  (Figure  13)  so  that  these  types  of  interdependencies  can 
be  better  understood. 
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Figure  15.  Flow  Diagram  for  the  Issues  Related  to  MDAP  A 

and  MDAP  C 
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X.  Conclusions  and  Future  Work 


We  have  conducted  a  case  study  of  the  MDAP_A  funding  network  based  on 
the  available  DAES  and  SAR  files  for  the  period  from  2004-2010.  Our  analyses  of 
these  disparate  yet  intrinsically  related  data  indicate  that  the  programs  are  related  to 
other  programs  based  on  funding  and  data  relationships.  This  supported  our  belief 
that  a  network-centric  predictive  model  would  be  a  good  candidate  for  MDAP 
performance  analyses.  We  have  also  noticed  that  although  the  available  data 
provide  useful  information  about  the  MDAPs,  it  is  challenging  to  integrate  and 
understand  these  data  coherently  such  that  network  dependencies  can  be  revealed 
accurately. 

We  then  observed  that  issues  that  led  a  program  towards  experiencing  an 
APB  breach  and/or  increase  in  %PAUC  were  not  solely  local  and  that  the  non-local 
issues  might  affect  the  performance  of  the  program.  We  studied  two  related 
programs,  MDAP_A  and  MDAP_B,  from  a  local  perspective  based  on  their 
respective  DAES  reports  and  showed  that  local  mitigation  efforts,  although 
successful  at  times,  still  resulted  in  APB  breaches  at  other  times.  Specifically,  we 
observed  from  the  SAR  files  that  the  cost  overrun  of  MDAP_A  in  2009  onwards 
might  have  affected  MDAP_B  in  2010  in  the  form  of  a  procurement  funding 
shortfall.  This  observation,  even  if  it  may  not  be  conclusive,  is  suggestive  of 
cascading  effects  between  neighboring  MDAPs.  Our  study  of  MDAP_B  in  2009  and 
2010  led  us  to  address  two  questions:  (1)  why  would  the  procurement  funding 
requirement  increase  in  2009  and  2010  and  (2)  what  is  the  reason  for  MDAP_B  not 
receiving  its  requested  amount  of  funding  that  resulted  in  a  funding  shortfall 
condition  for  two  consecutive  years?  Although  the  SAR  files  provide  an  answer  to 
the  first  question,  which  is  that  the  increase  in  quantity  led  to  the  need  for  increased 
funding,  our  data  did  not  provide  an  answer  to  the  second  question.  Hence  it 
appears  that  this  lack  of  knowledge  about  one’s  own  program  domain  (not  being 
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able  to  understand  the  root  cause  of  the  APB  breach  issues)  may  result  in  producing 
unexpected  cascading  effects  through  neighbor  programs. 


With  respect  to  a  formal  model,  we  first  argue  why  a  decision-theoretic  model 
based  on  MDPs  would  be  a  good  candidate  for  isolating  cascading  risks  for  the 
MDAP  network.  We  then  show  that  the  partially  observable  state  space  of  each 
program  warrants  a  DEC-POMDP  model,  which  belongs  to  the  class  of  MDPs.  The 
computational  complexity  of  the  original  DEC-POMDP  led  us  to  explore  feasible 
approximations  that  would  still  provide  the  performance  guarantees.  We  are 
currently  working  on  automating  this  decision-theoretic  model  for  the  nodes  in  the 
MDAP_A  funding  network. 

We  believe  that  true  joint  capability  relies  on  an  understanding  of  the  scope 
and  challenges  of  the  interdependencies  among  MDAPs.  Our  manual  analyses  of 
the  DAES  and  SAR  documents  for  a  focused  MDAP  case  study  reveal  indications 
about  possible  cascading  effects  and  offer  better  understanding  about  the  root 
causes  for  poor  performance  of  the  programs.  In  the  future,  we  plan  to  automate  this 
process  based  on  the  proposed  DEC-MDP  model,  leveraging  larger  data  sets.  It 
would  be  important  to  observe  how  the  second  and  higher  order  neighbors 
contribute  to  the  cascading  effects.  We  also  plan  to  extend  these  analyses  for  MDAP 
data  network  focusing  on  data  relationships  that  are  relatively  stable  over  multiple 
years. 


NAVAL  POSTGRADUATE  SCHOOL 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 


-62  - 


List  of  References 


Bernstein,  D.  S.,  Givan,  R.,  Immerman,  N.,  &  Zilberstein,  S.  (2002).  The  complexity 
of  decentralized  control  of  Markov  decision  processes.  Mathematics  of 
Operations  Research,  27(4),  819-840. 

Bertsekas,  D.  P.  (1987).  Dynamic  programming:  Deterministic  and  stochastic 
models.  Englewood  Cliffs,  NJ:  Prentice-Hall. 

Bethke,  E.  (2003).  Game  development  and  production.  Plano,  TX:  Wordware. 

Brown,  M.  M.  (201 1 ).  Acquisition  risks  in  a  world  of  joint  capabilities  (UNC-AM-1 1- 
162).  Retrieved  from  Naval  Postgraduate  School,  Acquisition  Research 
Program  website:  http://www.acquisitionresearch.net 

Brown  &  Owen.  (2012)  .Acquisition  risks  in  a  World  of  Joint  Capabilities:  A  Study  of 
Interdependency  Complexity  (UNC-AM-1 2-073).  Retrieved  from  Naval 
Postgraduate  School,  Acquisition  Research  Program  website: 
http://www.acquisitionresearch.net/  files/FY2012/NPS-AM-12-C9P15R02- 
073.pdf 

Cheng,  S.,  Raja,  A.,  &  Lesser,  V.  (in  press).  Multiagent  meta-level  control  for  radar 
coordination.  Web  Intelligence  and  Agent  Systems:  An  International  Journal. 
Retrieved  from  ftp://mas.cs.umass.edu/pub/WIAS_NetRads_Cheng.pdf 

Defense  Acquisition  Workforce  Act  of  1990,  10  U.S.C.  §  2434  (1990). 

Flowe,  R.  M.,  Brown,  M.  M.,  &  Hardin,  P.  L.  (2009).  Programmatic  complexity  and 
interdependence:  Emerging  insights  and  predictive  indicators  of  development 
resource  demand  (NPS-AM-09-058).  Retrieved  from  Naval  Postgraduate 
School,  Acquisition  Research  Program  website: 
http://www.acquisitionresearch.net 

Giambastiani,  E.  (2004,  March).  Imperatives  for  transformation.  Paper  presented  at 
ComDef  West,  San  Diego,  CA. 

Han,  S.  Y.,  Fang,  Z.,  &  DeLaurentis,  D.  (2012).  Acquisition  management  for  system- 
of-systems:  Requirement  evolution  and  acquisition  strategy  planning.  In 
Proceedings  of  the  Ninth  Annual  Acquisition  Research  Symposium  (Vol.  1, 
pp.  241-251).  Retrieved  from  http://www.acqusitionresearch.net 

Lewin,  Y.  (1999).  Application  of  complexity  theory  to  organization  science. 
Organization  Science,  70(3),  215-236. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-63- 


Newman,  M.  E.  J.  (2011).  Networks:  An  introduction.  New  York,  NY:  Oxford 
University  Press. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&LJ). 
(2008,  December  8).  Operation  of  the  defense  acquisition  system  (DoD 
Instruction  5000.02).  Washington,  DC:  Author. 

Zhao,  Y.,  Gallup,  S.,  MacKinnon,  D.  (2012).  Applications  of  Lexical  Link  Analysis 

Web  Service  for  Large-Scale  Automation,  Validation,  Discovery,  Visualization, 
and  Real-Time  Program-Awareness.  (NPS-AM-1 2-056).  Retrieved  from  Naval 
Postgraduate  School,  Acquisition  Research  Program  website: 
http://www.acquisitionresearch.net/  files/FY2012/NPS-AM-12-C9P07R01- 
056.pdf 


NAVAL  POSTGRADUATE  SCHOOL 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 


-64  - 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 

555  DYER  ROAD,  INGERSOLL  HALL 

MONTEREY,  CALIFORNIA  93943 


www.acquisitionresearch.net 


